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(57) A system (10) for regulating the flow of messages through a firewall (18) having a network protocol 
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VIRTUAL PRIVATE NETWORK ON APPLICATION GATEWAY 
5 fj Aground of the Invention 

FieH of fte Invention 

The present invention pertains generally to network communications, and 
in particular to a system and method for secufely transferring ^ 
between firewalls over an u&rotected network. 

10 Background Information 

FLewalls have become an increasingly important part of network design. 
Firewalls provide protection of valuable resources on a private network while 
allowing communication and access with systems located on en ur .ot':cted 
network such as the Internet. In addition, they operate to block attacks on a 

1 5 private network arriving from the unprotected network by providing a single 
connection with limited services. A well designed firewall limits the security 
problems of an Internet connection to a single firewall computer system. This 
allows an organization to focus their network security efforts on the definition of 
the security policy enforced by the firewall. An example of a firewall is given in 

20 "SYSTEM AND METHOD FOR PROVIDING SECURE INTL NETWORK 
SERVICES" by Boebert et al. (PCT Published Application No. Wi 96/131 13, 
published on May 2, 1 996), the description of which is hereby inct )orated by 
reference. Another description of a firewall is provided by Dsn Thomsen in 
"Type Enforcement: the new security model", Proceedings: Multimedia: Full- 

25 Service Impact on Business, Education, and the Home, SPIE Vol. 2617, p. 143, 
August 1996. Yet another such system is described in "SYSTEM AND 
METHOD FOR ACHIEVING NETWORK SEPARATION" by Gooderum et al. 
(PCT Published Application No. WO 97/29413, published on August 14, 1997), 
the description of which is hereby incorporated by reference. All the above 

30 systems are examples of application level gateways. Application level gateways 
use proxies or other such mechanisms operating at the application layer to 
process traffic through the firewall. As such, they can review not only the 



message traffic but also message content In rtidition, they provide 
authentication and identification services, access control and auditing. 

Data to be transferred on unprotected networks like the Internet is 
susceptible to electronic eavesdropping and accidental (or deliberate) corruption. 
Although a firewall can protect data within a private network from attacks 
launched from me unprotected network, even that data is vulnerable to both 
eavesdropping and corruption when transferred from the private network to an 
external machine. To address this danger, the Internet Engineering Task Force 
(IETF) developed a standard for protecting data transferred between firewalls 
over an unprotected network. The Internet Protocol Security (IPSEC) standard 
calls for encrypting data before it Irwes the first firewall, and then * -c-^nng 
the data when it is received by the second firewall. The decrypte ip !" x ' en 
delivered to its destination, usually a user workstation connected to ijs i ond 
firewall. For this reason IPSEC encryption is sometimes called fircwau ro- 
firewall encryption (FFE) and the connection between a workstation connected 
to the first firewall and a client or server connected to the second firewall is 
termed a virtual private network, or VPN. 

The two main components of IPSEC security are data ene^ition and 
sender authentication. Data encryption increases the cost and tm!F •quired for 
the eavesdropping party to read the transmitted data. Sender autba ° aiion 
ensures that the destination system can verify whether or not the CL^/pted data 
was actually sent from the workstation that it was supposed to be sent from. The 
IPSEC standard defines an encapsulated payload (ESP) as the mechtdsm used 
to transfer encrypted data. The standard defines an authentication header (AH) 
as the mechanism for establishing the sending workstation's identity. 

Through the proper use of encryption, the problems of eavesdropping and 
corruption can be avoided; in effect, a protected connection is established from 
the internal network connected to one firewall through to an internal network 
connected to the second firewall. In addition, IPSEC can be used to provide a 
protected connection to an external computing system such as a portable 
personal computer. 



IPSEC encryption and decryption work wrduh the IP layer of the network 
protocol stack. This means that all communication between two IP addresses 
will be protected because all mterfirewall communication must go through the IP 
layer. Such an approach is preferable over encryption and decryption at higher 
5 levels in the network protocol stack since when encryption is performed at layers 
higher than the IP layer more work is required to ensure that all supported 
communication is properly protected. In addition, since lPSEC encryption is 
Jiandled below the Transport kyer, il'SEC can encrypt data sent by any 
application. IPSEC therefore becomes a transparent add-on x» such protocols as 
TCP and UDP. 

Since, however, IPSEC decryption occurs at the IP layer, it can bi 
difficult to port IPSEC to an application level gateway while still main* jiing 
control at the proxy over authentication, message content, access coutul and 
auditing. Although the IPSEC specification in RFC 1825 suggests the use of a 
mandatory access control mechanism in a multi-level secure (MLS) ^u ^vbrk to 
compare a security level associated with the message with the security 1< vel of 
the receiving process, such ah approach provides only limited utility in n 
application levol gateway environment In fajt, implementations or 1 application 
level gateways to date have simply relied on the feet that toe message was 
IPSEC-encrypted as assurance that the message is legitimate and have simply 
decoded and forwarded the message to its destination. This creates, however, a 
potential chink in the firewall by assuming that the encrypted communication 
has access to all services. 

What is needed is a method of handling IPSEC messages within an 
application level gateway which overcomes the above deficiencies. The method 
should allow control over access by an IPSEC connection to individual services 
within the internal network. 

Summary nf the Tnventi^ 
The present invention is a system and method for regulating the flow of 
messages through a firewall having a network protocol stack, wherein the 
network protocol stack includes an Internet Protocol (IP) layer, the method 



ron^risiiig the steps of dstennming, at the IP layer, if a message is encrypted, if 
the message is not encrypted, passing the unencrypted message up the network 
protocol stack to an application level proxy, and if the message is encrypted, 
decrypting the message and passing the decrypted message up the network 
5 ■■ protocol stack to the application level proxy, wherein the step of decrypting the 
\message includes the step of executing a procedure at the .TP layer to decrypt the 
^message. 

According to another aspect of the present invention, a .system and 
method is described for authenticating the sender of a message within a 
0 computer system having a network protocol stack, wherein the network protocol 
stack includes an Internet Protocol (IP) layer, the method comprising the steps of 
determining, at the IP layer, if the message is encrypted, if the message is 
encrypted, decrypting the message, wherein the step of decrypting me message 
includes the step of executing a procedure at the IP layer to decrypt the message, 
passing the decrypted message up the network protocol stack to an spplicatipn 
level proxy, determining an authentication protocol appropriate for the message, 
and executing the authentication protocol to authenticate the sender of the 
message. 

Brief Description of the Drawings 
In the following detailed description of example embodiments of the 
invention, reference is made to the accompanying drawings which form a part 
hereof, and which is shown by way of illustration only, specific embodiments in 
which the invention may be practiced. It is to be understood mat otiwr 
embodiments may be utilized and structural changes may be made without 
departing from the scope of the present invention. 

In the drawings^ where like numerals refer to like components throughout 
the several views: 

Figure 1 is a functional block diagram of an application level gateway- 
implemented foewaU-to-nrewall encryption scheme according to the present 
invention; 



Figure 2 is a block diagram showing access control checking of both 
encrypted and unencrypted messages ir. network protocol stack according to the 
present invention; 

Figure 3 is a block diagram of a representative application level gateway- 
implemented firewall-to-firewall encryption scheme; 

Figure 4 is a block diagram of one embodiment of a network-separated 
protocol stack implementing IPSEC accbtb^ to the p^ent invention; and 

Figure 5 is a functional block diagram of a firewaU-to-workstation 
encryption scheme according to the present invention. 

Description of the Preferred Em^n^ 
In the following detailed description of the preferred embodiment, 
references made to the accompanying drawings which form a part hereof, and in 
which is shown by way of illustration specific preferred embodiments in which 
the invention may be practiced These embodiments are described in sufficient 
detail to enable those skilled in the art tc practice the invention, and it is to be 
understood that other embodiments may be utilized and that structural, logical, 
physical, architectural, and electrical changes may be made without departing 
from the spirit and scope of the present invention. The following detailed 
description is, therefore, not to be taken in a limiting sense, and the scope of the 
present invention is defined only by the appended claims and their equivalents. 

A system 10 which can be used for firewall-to-flrcwall encryption (FFE) 
is shown in Figure 1. In Figure I, system 10 includes a workstation 12 
communicating through a firewall 14 to an unprotected network 1 6 such as the 
Internet System 10 also includes a workstation 20 communicating through a 
firewall 18 to unprotected network 16. In one embodiment, firewall 1 8 is an 
application level gateway. 

As noted above, IPSEC encryption and decryption work within the IP 
layer of the network protocol stack. This means that all communications 
between two IP addresses will be protected because all interfirewall 
communication must pass through the IP layer. IPSEC takes the standard 
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Internet pack^a^convertefr^ The canier packet is 

designed to do two things: to conceal the contents of the original packet 
(encryption) and to provide a mechanism by which the receiving firewall can 
verify the source of the packet (authentication). In one embodiment of the 
present invention, each IPSEC carrier packet includes both an authentication 
header used to. authenticate the sending machine and an encapsulated payload 
containing encrypted data. The authentication header and the encapsulated 
payload features of IPSEC can, however/be used independently. As required in 
RFC 1 825, DES-CBC is provided for use in encrypting the encapsulated payload 
while the authentication header uses keyed MDS. 

To use IPSEC, you must create a security association (SA) for each 
destination TP address. In one embodiment, each SA contains the Mowing 
information: 

Security Parameters Index (SPI) - The index used to find a SA on 
receipt of an IPSEC datagram. 

Destination IP address - The address used to find the SA and 

trigger use of IPSEC processing on output, 

The peer SPI - The SPI value to put on a IPSEC datagram on 

output. 

The peer IP address - The destination IP address to be put into the 

packet header if IPSEC Tunnel mode is used. 

The Encryption Security Payload (ESP) algorithm to be used. 

The ESP key to used for decryption of input datagrams. 

The ESP key to used for encryption of output datagrams. 

The authentication (AH) dgorimm to be used. 

The AH key to be used for validation of input packets. 

The AH key to be used for generation of the authentication data 

for output datagr am . 

The combination of a given Security Parameter Index and Destination IP 
address uniquely identifies a particular "Security Association." In one 
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embodiment, the sending firewall usos the sending userid and Destination 
Address to select an appropriate Security Association (and hence SPI value). 
Hie receiving firewall uses the combination of SPI value and Source address to 
obtain the appropriate Security Association. 
5 A security association is normally one-way. An authenticated 

communications session between two firewalls will normally have two Security 
Parameter Indexes in use (one in each direction). TTie combination of a 
particular Security Parameter It4ax and a particular Destination Address 
uniquely identifies the Security Association. 

10 MoreMormationonthespecfficsofann'SECFFE 

be obtained from the standards developed by the IPSEC work group and 
documented in Security Architecture for IP (RFC 1825) and in RFCs 1 826- 
1829. 

When a datagram is received from unprotected network 1 6 or is to be 

15 transmitted to a destination across unprotected network 16, the fircwatl must be 
able to determine the algorithms, keys, etc. that must be used to process the 
datagram correctly. In one embodiment, this information is obtained via a 
security association lookup. In one such embodiment, the lookup routine is 
passed several arguments: iht source IP address if the datagram is being received 

2D from network 1 6 or the destination IP address if the datagram is to be transmitted 
across network 1 6, me SPI, and a flag that is used to indicate whether the lookup 
is being done to receive or transmit a datagram 

When an IPSEC datagram is received by firewall 18 from unprotected 
network 16, the SPI and source IP address are detennmeu. v y looking in the 

25 datagram. In one embodiment a Security Association Database (SADB) stored 
within firewall 18 is searched for the entry with a matching SPI, In one such 
embodiment, security associations can be set up based on network address as 
well as a more granular host address. This allows the network adnunistrator to 
create a security association between two firewalls with only a couple of lines in 

30 a configuration file on each machine. For such embaiiments, the entry in the 
Security Association Database that has both the matching SPI and the longest 
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address match is selected as the S A entry. In another such embodiment, each S A 
has a prefix length value associated with the address. An address match on a SA 
entry means that the addresses match for the number of bits specified by the 
prefix length value. 

5 There are two exceptions to this search process. First, when an S A entry 

is set marked as being dynamic it implies that the user of this SA may not have a 
fixed IP address. In this case ths match 5s My determined by the SPI value. 
Thus it is necessary that the SPi values for such SA entries fr* unique in the 
S ADB. The second exception is fcr S A entries marked as tunnel mode entries. 

10' In this case it is normally the case that the sending entity will hide its sowce 
address so that all that Is visible on the public wire is the destination address. In 
this case, like in the case where the SA entries are for dynamic IP addresses, the 
search is done exclusively on the basis of the SPL 

When transmitting a datagram across unprotected network 16 the SaDB 

15 is searched using only the destination address as an input In this case the entry 
which has the longest address match is selected and returned to She calling 
routine. 

In one embodiment, if firewall 1 8 receives datagrams which arc 
identified as either an IPJPROTOJPSECJBSP or IP_PROTOJ?SBC_AH 
20 protocol datagram, there must be a corresponding SA in the SADB or else 

firewall 1 8 will drop the packet and an audit message will be generated. Such an 
occurrence might indicate a possible attack or it might simply be a symptom of 
an erroneous key entry in the Security Associatr - ^-tabase. 

In a system such as system 10, application level gk y my firewall 1 8 acts 
25 as a buffer between unprotected network 16 and workstations such as 

workstation 20. Messages coming from unprotected network 1 6 are reviewed 
.and a determination is made as to whether execution of an authentication and 
.identification protocol is warranted. In contrast to previous systems, system 1 0 
also performs this same determination on IPSEC-enciypted messages. If 
ift desired, the same authentication and identification can be made on messages to 
\ be transferred from workstation 20 to unprotected network 16. Figure 2 



, \ ryA v — , -t-i «vi ^r*x*u u v^ubitbu iw wciciuiuic u ujs. awtuuc auturcss aaa on arc 

* associated with an SA. In the embodiment shown in Figure 2, an SADB Master 



illustrates one way of authenticating both encrypted and unencrypted messages 
in a system such as system 10. 

in the system of Figure 2 a network protocol stack 40 includes a physical 
layer 42, an Internet protocol (IP) layer 44, a Transport layer 46 and an 
5 application layer 48 . Such a protocol stack exists, for instance on application 
level gateway firewall 18 of Figure 1 , An application executing in application 
layer 48 can communicate to an application executing on another system by 
preparing a message and transmitting it through one of the existing transport 
services executing on transport layer 46. Transport layer 46 in turn uses a 
1 0 process executing in IP layer 44 to continue the transfer. Physical layer 42 
provides the software needed to transfer data through the communication 
hardware (e.g., a network interface card or a modem). As noted above, IPSEC 
executes within IP layer 44. Encryption and authentication is transparent to the 
host as long as the network administrator has the Security Association Database 
1 5 correctly configured and a key management mechanism is in place on the 
firewall. 

In application level gateway firewall 1 8, a proxy 50 operating within 
application layer 48 processes messages transferred between internal and 
external networks. All network-to-network traffic must pass through one of the 
20 proxies within application layer 48 before being the transfer across networks is 
allowed. A message arriving from external network 1 6 is examined at IP layer 
44 and an SADB is queried to determine if the source address and SPI are 



copy 52 is maintained in persistent memory at application layer 48 while a copy 
25 54 of SADB is maintained in volatile memory within the kernel. If the message 
is supposed to be encrypted, the message is decrypted based on the algorithm 
and key associated with the particular SA and the message is transferred up 
through transport layer 46 to proxy 50. Proxy 50 examines the source and 
destination addresses and the type of service desired and decides whether 
3 0 authentication of the sender is warranted. If so, proxy 50 initiates an 

authentication protocol The protocol may be as simple as requesting a user 
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name and password or it may include a challenge/response authentication 
process. Proxy SO also Sooks to see whether the message coining in was 
encrypted or not and may factor thai into whether a particular type of 
authentication is needed. In Telnet, for instance, user name/password 
5 authentication may be sufficient for an FFE link wiiL'e the security policy may 
dictate that a more stringent challenge/response protocol is needed for 
unencrypted links. In that case, proxy 50 will be a ?&' roxy and it will base 
its authentication protocol m whether the link was en-. ^> or _ot. 

Since IPSEC exe&iiec within IP layer 44 there 5s do need for uost 

10 firewalls to update their applications. Users: that already have IPSEC available 
on their own host machine will, however, have to request that the firewall 
administrator set up SA's in the SADB for their traffic. 

In the embodiment shown in Figure 2, a working copy 54 of the Security 
Association Database consisting of all currently active SA's is kept ^resident in 

1 5 memory for ready access by IP layer processing as datagrams are received and 
transmitted. In addition, a working master copy 52 of the SADB is maintained 
sn a file in nonvolatile memory. During system startup and initialization 
processing the content of all of the required SA's in master SADB 52 is added to 
the working copy 54 stored in kernel memory . 

20 In one embodiment, firewall 1 8 maintains different levels of security on 

internal and external network interfaces. It is desirable for a firewall to have 
different levels of security on both the internal and external interfaces. In one 
embodiment, firewall 18 supports three different levels, numbered 0 through 2. 
These levels provide a simple policy mechanism that controls percussion for 

25 both in-bound and out-bound packets. 

Level 0 - do not allow any in-bound or out-bound traffic unless there is a 
security association between the source and destination. 
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\ - Level 1 - Allow both in-bound and out-bound non-IPSEC traffic but 
\ force the use oflPSECifaSA exists for the address. (To support this firewall 
\ 18 must look for a SA for each b-bound datagram.) 

Level 2 - allow NULL security associations to exist, NULL associations 
5 arc just like normal security associations, except no encryption or authentication 
transform is performed on in-bound or out-bound packets that correspond to this 
NULL association* With Level 2 enabled, the machine will still receive 
unprotected traffic, but it will not transmit unless Level 1 is enabled. 

The default protection level established when the Security Association 
1 0 Database (S ADB) is initialized at boot time is 1 for in-bound traffic and 2 for 
out-bound traffic. 

An Access Control List, or ACL, is a list of rules that regulate the flow of 
Internet connections through a firewall These rules control how a firewall's 
servers and proxies will react to connection attempts, When a server or proxy 

15 receives an incoming connection, it performs an ACL check on that connection. 
An ACL check co mpares a set of parameters associated wife th e 
connection against a list o f ACL roles. The rules determine whether the 
connection is allowed or denied A rule can also have one or more side effects. 
A side effect causes the proxy to change its behavior in some fashion. For 

20 example, a common side effect is to redirect the destination IP address to an 
alternate machine. In addition to IP connection attempts, ACL checks can also 
made on the console logins and on logins made from serial ports. Finally, ACL 
checks can also be made on behalf of IP access devices, such as a Cisco box, 
through the use of the industry standard TACACS+ protocpl 

25 In one embodiment, the ACL is managed by an acid daemon running in 

the kernel of firewalls 10 and 30. The acid daemon receives two types of 
requests, one to query the ACL and one to administer it In one such 
embodiment, the ACL is stored in a relational database such as the Oracle 
database for fast access. By using such a database, query execution is 

30 asynchronous and many queries can be executing concurrently. In addition, 
these types of databases are designed to manipulate long lists of rules quickly 
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and efficiently. These qualities ensure that a given query cannot 
process that issued the query for any appreciable time (> 1-2 seconds). 

In one such embodiment, the database can hold up to 1 00,000 users and 
up to 10,000 hosts but can be scaled up to the capacity of the underlying 
5 database engine. The results of an ACL check is cached, allowing repeated 
checks to be turned arpqnd very quickly. ; 

Applications on firewalls 10 and 30 can query acid to detennineifa 
given connection attempt should be allowed to succeed. In cne embodiment the 
types of applications (i.e. "agents") that can make ACL queries cot be divided 
10 into four classes; 

1) Proxies. These allow connections to pass through firewall 10 or 30 in 

order to provide access to a remote service. They includr iuthp 

(authenticated telnet proxy), pftp (FTP proxy), httpp (HTTP proxy), and 

tcpgsp (TCP generic service proxy). 
15 2) Servers, These provide a service on the firewall itself They include ftpd 

andhttpd 

3) Login agents. Login agent is a program on the firewall flat can create a 
Unix shell. It is not considered a server because it cannot receive IP 
connections. One example is Aisr/bin/login when used to create a dialup 

20 session or a console session on firewall 1 0 or 30. Another example is the 

command srole. 

4) Network Access Servers (NAS). NAS is a remote IP access device, 
typically a dialup box manufactured by such companies as Cisco or 
Bridge. The NAS usually provides dialup telnet service and may also 

25 provide SLIP or PPP service. 

Proxies, servers, login agents, and NASes make queries to acid to 
determine if a given connection attempt should be allowed to succeed. All of the 
agents except NAS make their queries directly. NAS, because it is remote, must 
communicate via an auxiliary daemon that typically uses an industry standard 

30 protocol such as RADIUS or TACACS+. The auxiliary daemon (e.g., tacradd) 
in turn forwards the query, to local acid. 
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As a side effect of the query, acid tells the agent if authentication is 
needed. If no authentication is needed> the connection proceeds immediately. 
Otherwise acid provides (as another side effect) a list of allowed authentication 
methods thai the user can choos2 from. The agent can present a menu of choices 
5 or simply pick the first authentication method by default. Typical authentication 
methods incl^ jde plain password, SNK DSS, SDI SecuxID, LOCKout DES, and 
LOCKout IORTEZZA. In one embodiment, the list of allowed authentication 
methods varies depending on the host name, user name, time of day, or any 
combination thereof. 

10 In the case of a Level 0 policy, it would be safe to assume that all 

incoming traffic is encrypted or authenticated. In the case of Levels 1 through 2, 
a determination must be made whether or not a security association exists for a 
given peer. Otherwise an application may believe that in-bound traffic has been 
authenticated when it really has not, (That is why it is necessary to look for an 

15 SA on input of eachnon-IPSEC datagram.) 

In one embodiment, a flag which accompanies the message as it is sent 
from IP layer 44 to proxy 50 indicates whether the incoming message was or was 
not encrypted In another embodiment, proxy 50 accesses Security Association 
Database 54 (the table in the kernel can be queried via an SADB routing socket 

20 (PF-SADB)) to determine whether or not a security association exists for a given 
peer.. The SADB socket is much like a routing socket found in the stock BSD 
4.4 kernel (protocol family PF-ROUTE) except that PF-SADB sockets are used 
to maintain the Security Association Database (SADB) instead of die routing 
table. Because the private keys used for encryption, decryption, and keyed 

25 authentication are stored in this table, access must be strictly prohibited and 
allowed to only administrators and key management daemons. Care must be 
taken when allowing user-level daemons access to /dev/mem or /dev/kmem 

as well, since the keys arc stored in kernel memory and could be exposed with 
some creative hacking. 
30 In one embodiment, a command-line tool called sadb is used to support 

the generation and maintenance of in-kemei version 54 of SADB. The primary 
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interface between ibis tool and the SADB is the PF-SADB socket Tne kernel 
provides socket processing to receive client requests to add, update, or change 
entries in in-kemel SADB 34. As noted above, the default protection level 
established when the Security Association Database (SADB) is initialized at boot 
5 time is 1 for in-bound traffic and 2 for out-bound traffic. This may be changed 
by the use of the sadb command. 

The existing sadb command was derived from the NIST implementation 
of IP SEC As noted above, this tool is much like rout e in that it uses a special 
socket to pass data structures in and out of die kernel. There are three commands 
10 recognized by the sadb command: get, set, delete. The following simple shell 

script supports adding and removing a single SA entry to SADB 54. It shows 
one embodiment of a parameter order for adding a iSA to the SADB . 

# ! /bin/ah 
15 if [ $# -ne 1 1 
then 

echo "usage: $0 <on>j<off>" >&2 
exit 1 
fi . . 
20 ONOFF«$l 

addsa () 

{ 

IPADDRESS=$2 
25 PEERADDRESS-0. 0.0.0 

PREFIXLEN-0 # Nutn Of bits, 0 «> full 32 

bit match 

LOC^LADDRESS-O .0.0.0 
REALADDRESS-0 .0.0.0 
30 PORT»0 

PROTOCOLS 
UID-0 

DESALG=1 # Is DES-CBC 

IVLEN-4 # bytes 

35 DESKEY=0b0b0b0b0b0b0b0b 

DESKEYLEN-8 # bytes 

AHALG=1 # 1 = MDS 

AHKEY=30313233343536373031323334353637 

AHKEYLEN-16 # bytes 

40 LOCAL_SPI=$l 
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PEEir SPI=$1 
TUNNEL_MODE=0 . 
AHRESULTLEN=4' . 

C0MBINED_M0DE»1 . . # On output, 1 » ESP, then 

5 AH; 0 - AH, then ESP 
DYNAMIC_FLAG»0 

if [ n $ONOFF" - "on" 
• ■ : .then . 

10 V/sadb add da t $IPADDRESS SPREFIXLEN $LOCAL_SPi 

$UID $PEERADDR£SS $PEER__SPI $TUNN3L_MODE $LOCALADDRESS 
$REALADDRESS $P£OTOCOI. $PORT $DESALG $IVLEN $DESK2YLEN 
$DESKEY $DESKEYLEN SDESKEY $ AHALG $AHKEYLEN $AHKEY 
$AHKBYLEN 3AHKEY $AHRESULTLEN SCOMBINEDJ40DE 

15 $DYNAMIC_FLAG 

else . 

./sadb delete dst $IPADDRESS $LOCAL-SPI 
fi . ^ 
} 

20 

# Get down . to work : . 

addsa 500 172.17.128.115 # nuntber6.sctc.com 

The current status of in-kemel SADB 54 can be obtained with the sadb 
25 command. The get option allows dumping the entire SADB or a single entry. In 
one embodiment, the complete dump approach uses /dev/kmem to find the 
information. The information may be presented as follows: 



# sadb get dst ' 

30 

Local -SPI Address-Family Destination-Addr 
Pref ix_length UIp 

Peer-Address Peer-SPI Transport -Type 
Local -Address Real -Address 
35 Protocol Port 

ESP_Alg_ID ESP_IVEC_Length 

ESP_Enc_Key_length ESPJBnc_ESP_Key 
ESP__Dec_Key_length ESP__DecJBSP_Key 
. AH_Alg_ID AH~Data_Length ~ 
40 AH_Gen_Key_Length AH_Gen_Key 

AH__Check_Key_Length AH_Check_Key 
Combined_Tfode Dynamic_Flag 
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500 INET: number 6 . sctc . com 0 0 

0.0.0.0 500 Transport (0) 0 
5 0.0.0.0 0.0.0.0 

None None 

DES/CBC-RFC1829 (1) 4 

8 ObObObObObObObOb 

8 ObObObObObObObOb 
10 MDS-RPC182ft(l) 4 

16 30313233343536373031323334353637 

16 30313233343536373031323334353637 
ESP+AHU) 0 

501 INET: apokes.sctc.com 0 0 

15 0.0.0.0 501 Transport (0) 0 

o.o.o.o.o.o.o.o 
None None 

DES/CBC-RFC1829 (1) 4 

8 ObObObObObObObOb 
20 8 ObObObObObObObOb 

MD5-RFC1828(1)4 

16 30313232343536373031323334353637 

16 30313233343536373031323334353637 
ESP+AH(1) 0 

25 

End of list . 

'When a new entry is added to in-kemel SADB 54, the add process first 
checks to see that no existing entry will match the values provided in the new 
30 entry. If ho match is found then the entry is added to the end of the existing 
SADB list 

To illustrate the use and administration of an FFE, we'll go through an 
example using FFE 70 in Figure 3. Firewalls 14 and 18 are botii application 
level gateway firewalls implemented according to the p>>^mt invention. 
35 Workstations H2 and H3 both want to comraumratewithHl. Forthc 
administrator of firewalls 14 and 18, this is easy to accomplish. The 
: administrator sets up a line something like this (we'll only show the IP address 
part and SPI parts of the SA, since they're the trickiest values to configure. Also, 
assume that we are using tunnel mode): 

40 

# Hypothetical SW1 Config File 
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# 

# Fields are laid out in the following maimer: 

# ercaddrornet- IbcalSPI* peeraddr- peerSPI« 
realsrcaddra localaddr= keys 

5 

# The following entry sets up a tunnel between hosts 
behind SWl 

# and hosts behind SW2 , 

srcal72.16.0 o a.lqcalSPI»666 peer=192 • 168 .100 .5 , 
10 peerSPI«777 \ 

realsrcaddr«192; 1S8 . 100 . 5 localaddrs«0 • 0 . 0 „ 0 
key=Oxdeadbeeffadebabe 

Hypothetical SW2 Config Pile 

Fields are laid out in the following wanner: 
srcaddrorneta localSPI= peeraddr- peerSPI- 
realsrcaddr* localaddr* key« 

20 # The following entry" sets up a tunnel between hosts 
behind SWl and 

# hosts behind SW2 . 

src=172.17*0.0 localSPI*777 peer»192 . 168 , 20 . 1 
peerSPJ>66S \ 
25 realsrcaddral92. 168. 20. 1 localaddr=0 , 0 , 0 . 0 \ 

keyaOxdeadbeef f adebabe 

With this setup, all traffic is encrypted using one key, no matter who is 
talking to whom. For example, traffic from H2 to HI as well as traffic from H3 

30 to HI will be encrypted with one key. Although this setup is small and simple, it 
may not be enough* 

^ What happens if H2 cannot trust H3? In this case, the administrator can 
set up security associations at the host level. In this case, we have to rely on the 
SPI field of the SA, since the receiving firewall cannot tell from the datagram 

35 header which host behind the sending firewall sent the packet Since the SPI is 
stored in IPSEC datagrams, we can do a lookup to obtain its value, Below are 
the sample configuration files for both firewalls again, but this time, each host 
combination communicates with a different key. Moreover, H2 excludes H3 
from communications with HI, and H3 excludes H2 in the same way. 



# 

15 # 
# 
# 
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# Hypothetical SW1 Conf ig File 

: # 

# Fields are laid out in the following manner: 

# srcaddrornet= localSPI- peeraddr- peerSPI= 
5 realsrcaddr- localaddr- key- 

# The following entry sets up a secure link between H2. 
• and HI. 

src-172 . 16 .0.2 lbcalSPI"666 peer-192 .168 . 100 . 5 
10 peerSPI«777 \ 

realsrcaddr-192 . 16S . 100 . 5 
localaddrs-178.17.128.71 \ 

keyoOxOaOaOaOaOaOaOaOa 

15 # The following entry sets up a secure link between H3 
and HI 

src-172. 16. 0.1 localSPI-555 peer-192. 168. 100. 5 
peerSPI-888 \ 

realsrcaddr-192.168.100.5 
20 localaddrs-178.l7.12B.71 \ 
key=0x0b0b0bOb0b0b0bOb 

# Hypothetical SW2 Config File 
# 

25 # Fields are laid out in the following manner: 

# srcaddrornet- localSPI- peeraddr- peerSPI- 
realsrcaddr- localaddr- key- 

# The following entry sets up a secure link between H2 
30 and HI 

src-172. 17. 128. 71 localSPI-777 peer-192. 168. 20.1 

peerSPI-666 \ 

realsrcaddr-192. 168 .20.1 localaddrs-172 . 16 . 0 . 2 \ 

key-OxOaOaOaOaOaOaOaOa 

35 

# The following entry sets up a secure link between H3 
and HI 

src-172. 17. 128 . 71 localSPI-888 peer=192 . 168 . 20 .. 1 
peerSPI-555 \ 

40 realsrcaddr-192. 168. 2 0.1 localaddrs-172 . 16 . 0 . 1 \ 

key- OxObObObObObObObOb 

Figure 4 is a block diagram showing in more detail one embodiment of 
an EPSEC-cnabled application level gateway firewall 18. Application level 
45 gateway firewall 18 provides access control checking of both encrypted and 
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unencrypted messages in a more secure environment due to its network- 
scparatedj*rchitccture> Network separation divides a system into a set of 
independent regions or burbs, with a domain and a protocol stack assigned to 
each burb. Each protocol stack 40x has its own independent set of data 

5 structures, including routing information and protocol information. A given 
socket will be bound to a single protocol stack at creation time and no data can : 
pass between protocol stacks 40 withotrt gomg through proxy space. A proxy 50 
therefore acts as the go-between for transfers between domains. Because of this, 
a malicious attacker who gains control of om of the regions is prevrnted from 

1 0 being able to compromise processes executing in other regions. N' .work 
separation and its application to an application level gateway is de cn'xd in 
"SYSTEM AND METHOD FOR ACHIEVING NETWORK SfiPARAT10N , \ 
U.S. Application No. 08/599,232, filed February 9, 1 996 by Gooderum et aL 
In the system shown in Figure 4, the in-bound and out-bound datagram 

1 5 processing of a security association continues to follow the conventions defined 
by the network separation model. Thus all datagrams received on or Sent to a 
given burb remain in that burb once decrypted. In one such embodiment SADB 
socket 78 has been defined to have the type *sadb\ Each proxy 50 that requires 
access co SADB socket 78 to execute its query as to whether the received 

20 message was encrypted must have create permission to the sadb type* 

The following is list of specific requirements that a system such as is 
shown in Figure 4 must provide. Many of the requirements were discussed in 
the information provided earlier in this document 

1 . Firewall applications may query the IPSEC subsystem to determine if 
25 traffic with a given address is guaranteed to be encrypted. 

2. Receipt of an unencrypted datagram from an address that has a S A results 
in the datagram being dropped and an audit message being generated. 

3. On receipt of encrypted protocol datagrams the SADB searches will be 
done using the SPI as the primary key. The source address will a 

30 secondary key. The SA returned by the search will be the SA which 

matches the SPI exactly and has the longest match with the address. 
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4 A search of the SADB for a SPI that finds an entry that is marked as SA 
for a dynamic IP will not consider the address in the search process. 

5. Ase^ofihe-SADBfa'aSPI^^ineBlrytoiiiM^asaSA 
for a tunnel mode connection will to consider the address if it is (0.0.0.0) 
i.elNADDR. 

6. On receipt of a non-IPSEG datagram the SADB will be searched for an 
entry that matches the src address. If a SA is found the datagram will be 
dropped and an audit message sent 

7. SADB searches on output will be done using the DST address as key. If 
more than one SA entry in the SADB has that address the first one with 
the niaximum address match will be retumeci. 

8. The SADB must be structured so that searches are fast regardless if the 
search is done by SPI or by address. 

9. The SADB must provide support for connections to a site with a fixed 
5 SPI but changing IP address. SA entries for such connections will be 

referred to as Dynamic Address Sites, or just Dynamic entries. 

10. When a dynamic entry is found by a SPI search, the current datagram's 
SRC address, which is required to ensure that the return datagrams are 
properly encrypted, will be recorded in the SA only after the AH 

:0 checking has passed successfully. (This is because if the address is 

recorded before AH passes then an attacker can cause return packets of 
an outgoing connection to be transmitted in the dear.) 

11. %' A failure of an AH check oh a dynamic catty results in an audit message. 

12. In ah embodiment where the firewall requires that all connections use 
25 both AH and ESP, on receipt the order should be AH first ESP second. 

13. The processing structure on both input and output should try to minimize 
the number of SADB required lookups. 

Returning to FigurejynoManbjad^^ 
30 engine in terface 80 used to encrypt an IPSEC pay load. Crypto engine interface 
80 may be connected to a software encryption engine 82 or to a hardware 
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encryption engine 84. Engines 82 and 84 perform the actual encryption function 
using, for example, DES-CBC. In addition, software encryption engine 82 may 
include the keyed MD5 algorithm used for AH. 

In one embodiment, crypto engine interface 80 is a utility which provides 

5 a consistent interface between the software and hardware encryption engines. As 
shown in Figure 4, in one such embodiment interface 80 only supports me use of 
the use of hardware cryptoj-raphic engine 84 for IPSEC ESP processing. The 
significant tesign issue that interface 80 must deal with is mat xzse of a hardware 
encryption engine requires that the processing be down in disjoint steps 

10 operating in differem interrupt contexts as engine 84 completes the various 
processing steps. 

The required information is stored in a request strocture that is bound to 
the IP datagram being processed. The request is of type crvpto_rtsquest_t. 
This structure is quite large and definitely does not contain a rninimum state set. 
15 In addition to the definition of the request data structure, thus software 

implementing interface SO provides two functions which isolate the decision of 
which cryptographic engine to use. The crypt jies_encrypi: function is for 
use by the IP output processing to encrypt a datagram. The 
crypt_des_decrypt function is for use by the IP input processing to 
20 decrypt a datagram. If hardware encryption engine 84 is present and other 

hardware usage criteria are met the request is enqueued on a hardware processing 
queue and a return code indicating that me cryptographic processing is in 
progress is returned. If software engine 82 is used, the return code indicates that 
the cryptographic processing is complete. In the former case, the continuation of 
25 the IP processing is delayed until after hardware encryption is done. Otherwise 
it is completed as immediately in the same processing stream. 

There are two software cryptographic cngmes 82 prodded in the IPSEC 
software. One provides the MD5 algorithm used by the IPSEC AH processing, 
and the other provides the DES algorithm used by the IPSEC ESP processing. 
30 This software can be obtained from the US Government IPSEC implementation. 
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In one embodiment hardware cryptogenic engine 84 is provided by a 
Cylink SafeNode processing board. The interfile to this hardware card is 
provided by the Cylink device driver. A significant aspect of the Cylink card 

thatplaysamajor^ 
5 functions much like a low level subroutine interfile* and requires software 

support to uiitiate each tffocessmg step. Thus to encrypt or cecrypt an individual . 
datagram there are a minimum of two stcj)S, om to set the DES iaidalizadon 
vectw and one to do the encryption. Since the IP proc ssiog can not suspend 
itself and wait while the hardware completes and then be rescheduled by the 
10 hardware mterru^ 

tie seouences of hardware processing elements together. Inonesuch 
embodiment the interrupt handler looks at me current state, ese^^ 
after state function, transitions to the state and then executes that stale's start 
function. 

15 Onefur^en,cyl_en^ 

encrypt or a decrypt action. This function is designed to be caUed by 
cryptographic engine interface 80. All of the information required to initiate the 
processing as well as the function to be performed after the encryption operation 
is completed is provided in the request structure. This function will enqueue the 

20 request on the hardware request queue and start the hardware processing if 
necessary. 

A system 30 which can be used for firewall-to-workstation encryption is 
shown in Figure 5. In Figure 5, system 30 includes a workstation 12 
communicating through a firewall 14 to an unprotected network U such as the 

25 Internet System 30 also includes a workstation 32 ammiurucaung directiy with 
firewall 14 through unprotected network 16. Firewall 14 is an appUcation level 
gateway incorporating IPSEC handling as described above. (It should be noted 
thatlPSEC security cannotbeused to authenticate the personal identity of the 
sender for afirewall to firewall transfer. WhenlPSEC is used, however, on a 

30 single user machine such as aportable personal computer, IPSEC usage should 
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be protected with a personal idoitification number (PIN). In these cases IPSEC 

can be used to help with user identification to the firewall.) 

According to the IPSEC RFC -s, you can use either tunnel or transport 

mode with this embodiment based on your security needs. In certain situations, 
5 thecornmumcationsm^ 

Although specific embodiments have been Ulustrated and described 

herein, it will be appreciated by those of ordinary skill in the art that any 

arrangement which is calculated to achieve the same purpose may be substituted 

for the specific embodiment shown. This application is intended to cover any 
10 adaptations or variations of the present invention. Therefore, it is intended that 

this invention be limited only by the claims and the equivalents thereof . 
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What is claimed is: 

5 Protocol (IP) layer, the mated comprising the stepsof. 

defenaimog. at to IP layer, if a mess^eU encrypted; 
if (he message is not encrypted, passing the unencrypted message op the 
networkr«tocolstaclctoai.appUctti<«levelptoxyiaad 

10 decwtedmes^geuptonetworkprotoeolstacktoto^^^^ 

vvneremtostepofdeeryptmgtom 
prwedme at to IP layer to decrypt the message. 

2 A »emc4of»^catingtheseDderofamessne^thm 

15 ■^M*^V^^«*** v ~ m ** m * m * 

includes axlntemet Protocol (IP) layer, to nKthodcomprisiog to steps of: 

detennirdng, at to IP layer, if to message is encrypted; 
if fl* message is encrypted, decking to message, therein to step of 
decrypting to message includes to sup of executing a process a, to IP layer to 

20 decrypt the message; 

passing the decrypted message up the network protocol stack to an 

application level proxy, 
•> fctenmmr*anau^ 

executing the authentication protocol to aumenticate ^sender of the 

25 message. 

3. The method according to claim 2 wherein the step of determimng an 
authentication protocol appropriate for the message includes . the steps of: 
detenrurung a source IP address associated with the message; and 
30 detentuning the authentication protocol associated with the source IP 

address. 
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4. The method according to claim 2 wherein the message includes security 
parameters index and wherein the step of deterniining an authentication protocol 
appropriate for the message incudes the steps of: 
1 determining the authentication protocol associated with a dynamic IP 

S address, wherein the step of detcnnir^ the au^ includesthe 
1 step of looking up a security ssrociation based cu the security parameters index; 
determining a curre^ addieu associated v«4 axe *mainic source IP . 

address; and y ' 

binding the current address to the security parameters index. 
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5. A firewall, comprising: 

a first communications interface; 
a second communications interface; 
a network protocol stack connected to the first and the second 
communiwtionsmteTfa^, whereto 

Internet Protocol (IP) fey* 8 trans P ort 

a decryption procedure, operating at the IP layer, wherein the decryption 
- procedure decrypts encrypted messages received at one of said first and second 
communications interfaces and outputs o^rypted messages; and 
20 aproxy, connected to the transport layer of said network protocol stack, 

wherein the proxy receives decrypted messages from the decryptloix procedure 
and executes an authentication protocol based on the content of the decrypted 
message. 

25 6. A firewall, comprising: 

a first communications interface; 
a second communications interface; 

a first network protocol stack connected to the first communications 
mterface, wherein the first network protocol stack includes an Internet Protocol 
30 (IP) layer and a transport layer; 
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a second network protocol stack connected to fhesecond 
communications interface, wherein the second network protocol stodc includes 
an Internet Protocol (IP) layer and a transport layer, 

a decryption procedure, operating at the IP layer of the first network 
5 protocol stack, the decryption procedure receiving encrypted messages received 
by said first cornmunications interface tad outputting decrypted messages; and 

a proxy, connected to the transport layers of said first and second network 
protocol stacks, the proxy r living decrypted messages from the deer i : uon 
procedure, and executing to authentication protocol based on the con^t of the 
10 decrypted message. 

7. The firewall according to claim 6 wherein the firewall further includes: 
a third communications interface; and 

a third network protocol stack connected to the third comraunications 
15 interface and to the proxy, wherein the third network protocol stack 1m^ 
Internet Protocol (IP) layer and a transport layer and wherein the second and 
third network protocol stacks are restricted to first and 3 econd bmbs, 
respectively. 

20 8. A method pf establish^ 

second network, wherein each network includes an application level gateway 
firewall which uses a proxy operating at the application layer to process traffic 
through the firewall, wherein each firewall includes a network protocol stack and 
wherein each network protocol stack includes an Internet Protocol (IP) layer, the 

25 method comprising the steps of: 

transferring a connection request from the first network to the. second 

network; 

determining, at the IP layer of the network protocol stack of the second 
network's firewall, if the connection request is encrypted; 
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if the connection request is encrypted, ^deaypting the request, wherein the 
step of decrypting the request includes the step of executing a procedure ^ the IP 
layer of the second network's firewall to decrypt the message; % 

passing the connection request up the network protocol stack to an 

5 application level proxy; 

determining an authentication protocol appropriate for the connection 

request; 

executing the aumeoticsiion protocol to auth^tjeate me connection 
request; and 

10 if the connection request is authentic establishing an active connection 

between the first and second networks. 

9. The method according to claim 8 wherein the step of executinr me 
authentication protocol includes the step of exectm^g program code within the 

15 firewall of the second network to mimic a challenge/response protocol executing 
on a server internal to the second network. 

10. The method according to claim 8 wherein the step of executing the 
authentication protocol includes the step of executing program code to execute 

20 the authentication protocol in line to the session. 

K . ' ' ■ • 

11. Thernemodaccordir^toc^ 

authentication protocol includes the step of determining.if the connection request 
arrived encrypted and selecting the authentication protocol based on whether the 
25 connection request was encrypted or not encrypted. 
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